Application of reinforcement learning to balancing of acrobot

The acrobot is a two-link robot, actuated only at the joint between the two links. It is one of dicult tasks in reinforcement learning (RL) to control the acrobot because it has nonlinear dynamics and continuous state and action spaces. In this article, we discuss applying the RL to the task of balancing control of the acrobot. Our RL method has an architecture similar to the actor-critic. The actor and the critic are approximated by normalized Gaus-sian networks, which are trained by an on-line EM algorithm. We also introduce eligibility traces for our actor-critic architecture. Our computer simulation shows that our method is able to achieve fairly good control with a small number of trials. 1. INTRODUCTION Humans are able to acquire control of their bodies through trial-and-error without detailed knowledge about body dynamics. Reinforcement learning (RL) is a kind of machine learning scheme that is like this. RL methods have been successfully applied to various Markov decision problems that have nite state/action spaces, such as the backgammon game [1]. On the other hand, applications to control problems of human or robot motion are much more dicult because the state and action spaces in these systems are continuous. In such cases, a good function approximator and a fast learning algorithm are crucial to achieving good performance. In our previous paper [2], we proposed an RL method based on our previously developed on-line EM algorithm [3] and applied it to a couple of control problems in which the state and action spaces were continuous. In this article, we discuss applying this method to the task of how to control the balance of an acrobot [4, 5] that is a two-link underac-tuated robot roughly analogous to a gymnast swinging on a high bar. Our model has an architecture similar to the actor-critic model proposed by Barto et al [6]. The actor yields a control signal for the current state, and the critic predicts the accumulation of rewards given in the future. However , in our model, the detailed implementations of the actor and the critic are quite dierent from those of the original model. The actor learns a control that maximizes the output of the current critic. The critic learns the Q-function that follows the current actor, and is based on the Bellman's equation [7]. The actor and the critic are then approximated by the normalized Gaussian networks (NGnet's) [8], which are …